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Model-based testing (MBT) is an well-known technology, which allows for automatic test case gen¬ 
eration, execution and evaluation. To test non-functional properties, a number of test MBT frame¬ 
works have been developed to test systems with real-time, continuous behaviour, symbolic data and 
quantitative system aspects. Notably, a lot of these frameworks are based on Tretmans’ classical 
input/output conformance (ioco) framework. However, a model-based test theory handling proba¬ 
bilistic behaviour does not exist yet. Probability plays a role in many different systems: unreliable 
communication channels, randomized algorithms and communication protocols, service level agree¬ 
ments pinning down up-time percentages, etc. Therefore, a probabilistic test theory is of great prac¬ 
tical importance. We present the ingredients for a probabilistic variant of ioco and define the pioco 
relation, show that it conservatively extends ioco and define the concepts of test case, execution and 
evaluation. 


1 Introduction 

Model-based testing (MBT) is a way to test systems more effectively and more efficiently. By generating, 
executing and evaluating test cases automatically from a formal requirements model, more tests can be 
executed at a lower cost. A number of MBT tools have been developed, such as the Axini test manager, 
JTorx n], STG Q, TorXakis IITSl . Uppaal-Tron [[101 US, etc. 

A wide variety of model-based test theories exist: the seminal theory of Input/Output conformance 
ll25l |27l is able to test functional properties, and has established itself as the robust core with a wide 
number of extensions. The correct functioning of today’s complex cyberphysical systems, depends not 
only on functional behaviour, but largely on non-functional, quantitative system aspects, such as real¬ 
time and performance. MBT frameworks have been developed to support these aspects: To test timing 
requirements, such as deadlines, a number of timed ioco-variants have been developed, such as IH [TOl 
[T5l . Symbolic data can be handled by the frameworks in (lllTdl; resources by Ell, and hybrid aspects in 

m- 

This paper introduces pioco, a conservative extension of ioco that is able to handle discrete proba¬ 
bilities. Starting point is a requirements model as a probabilistic quiescent transition system (pQTS), an 
input/output transition system, with two additional features: (1) Quiescence, which models the absence 
of outputs explicitly via a distinct 5 label: quiescence is an important notion in ioco, because a system- 
under-test (SUT) may fail a certain test case given an output is required, but the SUT does not provide 
one. (2) Discrete probabilistic choice. We work in the input-generative / output-reactive model |5*l, 
which extend Segala’s classical probabilistic automaton model EOll : upon receiving an input, a pQTS 
chooses probabilistically, which target state to move to. For outputs, a pQTS chooses probabilistically 
both which action to take, and which state to move to, see Figure[T]for an example. 

An important contribution of our paper is the notion of test case execution and evaluation. In partic¬ 
ular, we show how the use of statistical hypothesis testing can be exploited to determine the verdict of a 
test execution: if we execute a test case sufficiently many times and the observed trace frequencies do not 
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coincide with the probabilities described in the specification pQTS depending on a predefined level of 
significance, then we fail the test case. In this way, we obtain a clean framework for test case generation, 
evaluation and execution. However, being a first step, we mainly establish the theoretical background. 
Further Research is needed to implement this theory into a working tool for probabilistic testing 

Related work. An early and influential paper on probabilistic testing is Bisimulation Through Proba¬ 
bilistic Testing IHtII . which not only defines the fundamental concept of probabilistic bisimulation, but 
also shows how different (i.e. non-bisimilar) probabilistic behaviours can be detected via statistical hy¬ 
pothesis testing. This idea has been taken further in our earlier work l|4j|22l, which shows how to observe 
trace probabilities via hypothesis testing. 

Testing probabilistic Finite State Machines is well-studied (e.g. lITSll ) and coincidences to ioco theory 
can be found. However pQTS are more expressive than PFSMs, as they support non-determinism and 
underspecification, which both play a fundamental role in testing practice. Hence, they provide more 
suitable models for today’s highly concurrent and cyberphysical systems. 

A paper that is similar in spirit to ours is by Hierons et al. ifTTlfT^ . and also considers input reactive 
/ output generative systems with quiescence. However, there are a few important differences: Our model 
can be considered as an extension of |[TT1l reconsiling probabilistic and nondeterministic choices in a 
fully fledged way. Being more restrictive enables ifTTlfT^ to focus on individual traces, whereas we use 
trace distributions. 

Other work that involves the use of probability is given in l|71|28l|29l, which models the behaviour of 
the tester, rather than of the SUT as we do, via probabilities. 

Organization of the paper. We start by defining overall preliminaries in SectionSection [^defines 
the conformance relation pioco for those systems and Section provides the structure for testing and 
denotes what it means for an implementation to fail or pass a test suite by the means of an output and a 
statistical verdict. The paper ends with conclusions and future work in Section 

2 Probabilistic quiescent transition systems 

2.1 Basic definitions 

Definition 1. (Probability Distribution) A discrete probability distribution over a set A is a function 
p : X —)• [0,1] such that Lxex p{x) = 1. The set of all distributions over A is denoted as Distr{X). The 
probability distribution that assigns 1 to a certain element x G A is called the Dirac distribution over x 
and is denoted Dirac (x). 

Definition 2. (Probability Space) A probability space is a triple ,¥), such that D is a set, is a 

a-field of Tl, and P : [0,1] a probability measure such that P (fl) = 1 and P (UHo'^;') = 

for A,-, / = 1,2,... pairwise disjoint. 

2.2 Probabilistic quiescent transition systems 

As stated, we consider probabilistic transitions that are input reactive and output generative 1*9]: upon 
receiving an input, the system decides probabilistically which next state to move to. However, the system 
cannot decide probabilistically which inputs to accept. For outputs, in contrast, a system may make a 
probabilistic choice over various output actions. This means that each transition in a pQTS either involves 
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a single input action, and a probabilistic choice over the target states; or it makes a probabilistic choice 
over several output actions, together with their target states. We refer to Figure[^for an example. 

Moreover, we model quiescence explicitly via a 5-label. Quiescence means absence of outputs and 
is essential for testing: if the SUT does not provide any outputs, a test must determine whether or not this 
behaviour is correct. In the non-probabilistic case, this can be done either via the suspension automaton 
construction ll2^ . or via QTSs ll23l . The SA construction involves determinization. However, this is 
an ill-defined term for probabilistic systems. Therefore, we use the quiescent-labelling approach and 
demand to make quiescence explicit. 

Finally, we assume that our pQTSs are finite and don’t contain internal steps (i.e., T-transitions). 

Definition 3. (pQTS) A probabilistic quiescent transition system (pQTS) is an ordered five fuple = 
[S,so,Li,Lq,A) where 

• S a finife sef of sfafes, 

• So £ S fhe inifial sfafe, 

• Lj and Lq disjoinf sefs of inpuf and oufpuf acfions, wifh af leasf 5 G Lq. We wrife L := LjVJLq for 
fhe sef of all labels and lef Lq = Lq\ {5} fhe sef of all real oufpufs. 

• AC S X Distr (LxS) a finife fransifion relation such fhaf for all ( 5 , ft) G A, al £ Li,b £ L, s' ,s'' £ S, 
if ft {a2,s') > 0, then ft {b,s") = 0 for all b / al. 

We write s^ s' if ( 5 ,ft) G A and ft {a,s')> 0; and 5 —)■ a if there are p. £ Distr {L x S) and s' £ S such 
that s ^ s'. If it is not clear from the context about which system we are talking, we will write 5 s', 

{s,p)^ and s a to clarify ambiguities. Lastly we say that £/ is input enabled if for all 5 G 5 we have 
s ^ al for every a £ Lj. 

2.3 Paths and traces 

We define the usual language-theoretic concepts for pQTSs. 

Definition 4. Let js/ = {S,so,Li,Lq,^^ be a pQTS. 

• A path 71 of a pQTS £/ is a (possibly) infinite sequence of the form 


71 = siPiaiS2P2a2S3P3asS4...., 

where st G S, a, G L for / = 1,2,... and ft G Distr {L,S), such that each finite path ends in a state 
and Si for each nonfinal i. We use the notation first (n) := to denote the first state of 

a path, as well as last{n) := Sn for a finite path ending in 5„, and last{7i) = 0 ° for infinite paths. 
The set of all finite paths of a pQTS £/ is denoted by Path* ( 32 /) and the set of all infinite paths by 
Path {£/) respectively. 

• The trace of a path tt = sipiaiS 2 P 2 as 3 ... is the sequence obtained by omitting everything but the 

action labels, i.e. trace (tt) = aia 2 aj, _ 

• All finite traces of £/ are summarized in traces ( 32 /) = {trace (tt) G L* | tt G Path* {£/)}. 

• We write Sn with a £ L* for G S in case there is a path tt = i'lfiiai.. .Pn-tan-\Sn with 

trace (tt) = a and si for / = 1,..., n — 1. 

• We write reach^ {S', a) for the set of reachable states of a subset S' CS via a, i.e. 

reach^ [S', a) = G S | 3^' G S' : / . 
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• All complete initial traces of £/ are denoted by ctraces (^), which is defined as the set 

{trace{it) \ 7t G Path{^) :first{jt) = ^o, |7r| = oo VVa G L : reach^ {last{Ti) ,a) = 0} . 

• We write after^ (5) for the set of actions, enabled from state s, i.e. after^ (5) = {a G L | ^ a}. 

We lift this definition to traces by defining 

after^ (a) = [J after^ (s). 

sEreachj^{s(),G) 

• We write out^^ (a) = after (a) DLq to denote the set of all output actions as well as quiescence 
after trace a. 

In order for a pQTS to be meaningful, Il2^ postulated four well-formedness rules about quiescence, 
stating for instance that quiescence should not be succeeded by an output action. Since our current 
treatment does not rely on well-formedness, we omit these rules here. Moreover, our definition of a test 
case is a pQTS that does not adhere to the well-formedness criteria. 

2.4 Trace distributions 

Very much like the visible behaviour of a labelled transition system is given by its traces, the visible 
behaviour of a pQTS is given by its trace distributions: each trace distribution is a probability space that 
assigns a probability to (sets of) traces Il20]| . Just as a trace in an LTS is obtained by first selecting a path 
in the LTS and by then removing all states and internal actions, we do the same in the probabilistic case: 
we first resolve all the nondeterministic choices in the pQTS via an adversary, and by then removing all 
states — recall that our pQTSs do not contain internal actions. The resolution of the nondeterminism 
via an adversary leads to a purely probabilistic structure where we can assign a probability to each finite 
path, by multiplying the probabilities along that path. The mathematics to handle infinite paths is more 
complex, but completely standard |]6|: in non-trivial situations, the probability assigned to an individual 
trace is 0 (cf., the probability to always roll a 6 with a dice is 0). Hence, we consider the probability 
assigned to sets of traces (e.g., the probability that a 6 turns up in the first 100 dice rolls). A classical 
result in measure theory shows that it is impossible to assign a probability to all sets of traces. Therefore, 
we collect those sets that can be assigned a probability in a so-called a-field 

Adversaries. Following the standard theory for probabilistic automata ||2T1 . we define the behavior of 
a pQTS via adversaries (a.k.a. policies or schedulers). These resolve the nondeterministic choices in 
pQTSs: in each state of the pQTSs, the adversary chooses which transition to take. Adversaries can 
be (1) history-dependent, i.e. the choice which transition to take can depend on the full history; (2) 
randomized, i.e. the adversary may make a random choice over all outgoing transitions; and (3) partial, 
i.e., at any point in time, a scheduler may decide, with some probability, to terminate the execution. 

Thus, given any finite history leading to a current state, an adversary returns a discrete probability 
distribution over the set of available next transitions (distributions to be precise). In order to model 
termination, we define schedulers which continue the transitions of pQTSs with a halting extension. 

Definition 5. (Adversary) A (partial, randomized, history-dependent) adversary £ of a pQTS = 
{S,so,Li,Lo,A) is a function 


E : Path* (^Z) —)> Distr (Distr (L x S) U {-L}) 
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such that for each finite path n, if E (n) (/r) >0, then {last {n ), /r) G A. The value E {n) (_L) is considered 
as interruption/halting. We say that E is deterministic, if E {%) assigns the Dirac distribution for every 
distribution after all n G Path* (^). An adversary E halts on a path n, if it extends n to the halting state 
_L, i.e. 

E{k){1.) = 1. 

We say that an adversary halts after G N steps, if it halts for every path n with \ n\ >k. We denote all 
such adversaries by Adv {s^,k). Lastly E is finite, if there exists ^ G N such that E G Adv {£/,k). 

The probability space assigned to an adversary. Intuitively an adversary tosses a coin at every step 
of the computation, thus resulting in a purely probabilistic (as opposed to nondeterministic) computation 
tree. 

Definition 6. (Path Probability) Let E be an adversary of . The function : Path* (j 2 /) —>■ [0,1] is 
called the path probability function and it is defined by induction. We sef (sq) = 1 and {npas) = 
{n)-E{n){p)-p{a,s). 

Loosely speaking, we follow a finile pafh in fhe fransifion sysfem and mulfiply every scheduled prob- 
abilify along fhe way, resolving every nondeferminism according fo fhe adversary E fo gel fhe ultimate 
pafh probabilily. The pafh probabilily function enables us fo define a probabilily space associated wilh 
an adversary, Ihus giving every pafh in a pQTS an exacl probabilily. 

Definition 7. (Adversary Probability Space) Let E be an adversary of The unique probability space 
associated to E is the probability space {Q.e,^e,Pe) given by. 

1. Q.E = Path°° {£/) 

2. is the smallest a-field that contains the set {Cjc \ n G Path* where the cone is defined as 
Cji = {it' G Dg I 71 is a prefix of n'}. 

3. Pe is the unique probability measure on ^e s. t. Pe {Cf) = {tl), for all 7l G Path* {s^). 

The set of all adversaries is denoted by adv {saf) with adv {s^ ,k) being the set of adversaries halting after 
k G N respectively. 

Trace distributions. As we mentioned, a trace distribution is obtained from (the probability space 
assigned to) an adversary by removing all states. This means that the probability assigned to a set of 
traces X is defined as the probability of all paths whose trace is an element of X. 

Definition 8. (Trace Distribution) The trace distribution El of an adversary E, denoted // = trd {E) is 
the probability space {Q.h,^h,Ph) given by 

1- ^h=L*^ 

2. is the smallest a- field containing the set {C^ | j8 G where the cone is defined as Cp = 
{P' G Df I j8 is a prefix of j8'} 

3. Ph is the unique probability measure on such that Ph{X) = Pe {trace^^ (X)) for X G ^h- 

As an abbreviation, we will write Ph{^) ■= Ph {Cp) for j3 G L*^ 

Like before, we denote the set of trace distributions based on adversaries of £/ by trd{£/) and 
trd {aaf ,k) if it is based on an adversary halting after k G N steps respectively. Lastly we write =td ^ 
if trd (j^/) = trd £/ [Zye) dd if trd {£/) C trd (=^) and £/ if trd {j^,k) C trd {.'3S, k) for k G N, 
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Figure 1: An example of the combination of nondeterministic and probabilistic choices. 

where the embedding means that for every trace distribution H of £/ there is a trace distribution H' of ^ 
such that for all traces a of we have Ph{<^) = Pw (<7)- 

The fact that {flE,,^E,PE), {^h,^h-,Ph) really define probability spaces, follows from standard 
measure theory arguments (see Il6ll. 

Example 9. Consider the pQTS = [S,so-Li,Lq,A) inFigurej^ There 5 = • • Aio}> Pi = {«?}> 

= {f7!,c!,r/!}u{5} and A = {(5o,/io,), (^0,^02) , (■^0,^03) , (■^i,Mi)(i'io,Atio)}- We can see that 
this system has both probabilistic and nondeterministic choices. Observe that it has indeed only input 
reactive and output generative transitions as mentioned in the beginning of 2.2 

We will now consider an adversary E for . The only nondeterministic choice we have in this 
system, is located at state ^o, where we can either apply a? to enter the left branch, a? to enter the right 
branch, or do nothing (corresponding to Moij M02 M03 respectively). Therefore consider the adversary 

E (so) (MOi ) = 2 P (‘^0) (M02) = 2 E (it) (m) = Dirac for every other distribution p after a path it 
(i.e. those are taken with probability 1). 

The adversary probability space created for this adversary assigns an unambiguous path probability 
to each path. Consider the path it = SQPQ^als\P\b\s^, then 



However, consider the trace distribution El = trd(E). Then for a = alb\, we have trace ' (a) = {?r, t]} 
with 7t as before and rj = soPo-,‘^'^^3P3p^-^8- Hence 



3 The probabilistic conformance relation pioco 

3.1 The pioco relation 

The classical input-output conformance relation ioco states that an implementation £// conforms to a 
specification 32 /s if -s/i never provides any unspecified oufpuf. In parficular fhis refers fo fhe observation 
of quiescence, when ofher oufpuf was expecfed. 
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Figure 2: An example illustrating pioco 


Definition 10. (Input- Output Conformance) Let and be two QTS and let be input enabled. 
Then we say £/[ C,oco if and only if 

Va G traces ■ outj^. (a) C outj^^ (a). 

To generalize ioco to pQTSs, we introduce two auxiliary concepts. For a natural number k, the prefix 
relation H H' states that trace distribution H assigns exactly the same probabilities as H’ to traces of 
length k and halts afterwards. The output continuation of a trace distribution H prolongs the traces of H 
with output actions. More precisely, output continuation of H wrt length k contains all trace distributions 
that (1) coincide with H for traces upto length k and (2) the k + 1st action is an output label (incl 5); 
i.e. traces of length k + \ that end on an input action are assigned probability 0. Recall that Ph (cj) 
abbreviates Pu (Ca). 

Definition 11. (Notations) For a natural number k gN, and trace distributions H G trd{£/,k), we say 
that 

1. H is a prefix of PI' G trd {s^) up to k, denoted by H H', iff Va G : P// (a) = Ph' (a). 

2 . the output continuation oiH in is given by 

outcont{H,x 2 f,k) : = |//'G 1) |////'A Va G L^L/: P/^. (a) = o|. 

We are now able to define the core idea of pioco. Intuitively an implementation should conform to a 
specification, if the probability of every trace in specified in £/s, can be matched in the specification. 
Just as in ioco, we will neglect underspecified traces continued with input actions (i.e., everything is 
allowed to happen after that). However, if there is unspecified output in the implementation, there is at 
least one adversary that schedules positive probability to this continuation, which consequently cannot 
be matched of output-continuations in the specification. 

Definition 12. Let and be two pQTS. Furthermore let £/i be input enabled, then we say i^pioco 
s^s if and only if 


'ik G NV// G trd {^s,k) : outcont {H,^i,k) C outcont{H,£/s,k) ■ 

Example 13. Consider the two systems of ssf and shown in Figure and assume that p G [0,1]. 
It is true that £/ i^pioco because we can always choose an adversary E of PS, which imitates the 
probabilistic behaviour of PS, i.e. choose E{e){p) = V such that v (fl:!,ti) = p and v {b\,t 2 ) = 1— p. 

However, the opposite does not hold. For example assume p = \, then the trace distribution H 
assigning Pu (a!) = 1 is in outcont {H, SB, 1) but not in outcont {H, sP , 1) and hence M'^pioco^■ 
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3.2 Properties of the p-ioco relation 

As stated before, the relation pioco conservatively extends the ioco relation, i.e. both relations coin¬ 
cide for non-probabilistic QTSs. Moreover, we show that several other characteristic properties of ioco 
carry over to pioco as well. Below, a QTS is a pQTS where every occurring distribution is the Dirac 
distribution. 

Theorem 14. Let and be two QTS and let be input enabled, then 

^ioco '' '' ^=pioco ■Sfs- 

Intuitively it makes sense that the implementation is input enabled, since it should accept every input 
at any time. The following two results justify, that we assume the specification to be not input enabled, 
since otherwise pioco would coincide with trace distribution inclusion. Equivalently it is known that ioco 
coincides with trace inclusion, if we assume both the implementation and the specification were input 
enabled. Thus, as stated before, we can see that pioco extends ioco. 

Lemma 15. Let and £/s be two pQTS, then 

["7D ^pioco 

Theorem 16. Let s^i and s^s be two input enabled pQTS, then 

^pioco ' ' ' ' ErO 

Next, we show that, under some input-enabledness restrictions, the pioco relation is transitive. Again, 
note that this is also true for ioco for non-probabilistic systems. 

Theorem 17. (Transitivity of pioco) Let and if be pQTS, such that and fd are input enabled, 

then 


^ '=pioco 


^pioco ■ 


g/ C • 

'=pioco 




4 Testing for pQTS 

4.1 Test cases for pQTSs. 

We will consider tests as sets of traces based on an action signature , which will describe possible 

behaviour of the tester. This means that at each state in a test case, the tester either provides stimuli or 
waits for a response of the system. Additionally to output conformance testing like in Il24l . we introduce 
probabilities into our testing transition system. Thus we can represent each test case as a pQTS, albeit 
with a mirrored action signature {Lo,Lj U {5}). This is necessary for the parallel composition of the test 
pQTS and the SUT. 

Since we consider tests to be pQTS, we also use all the terminology introduced earlier on. Addition¬ 
ally we require tests to not contain loops (or infinite paths respectively). 

Definition 18. A test (directed acyclic graph) over an action signature (^Li,Lq) is a pQTS of the form 
1 = {S,so,Lo,LjLI{5} ,A) such that 

• t is internally deterministic and does not contain an infinite path; 

• t is acyclic and connected; 
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Figure 3: A specification for a simple shuffle music player and a test. 


• For every state s G S, we either have 

- after (s) = 0 , or 

- after (s) =L/U{5}, or 

- after {s) = {a!} UL/ U {5} for some a\ G Lq- 


A test suite T is a set of tests over an action signature {Lj,L^ . We write .‘f {Li,L^ to denote all the tests 
over an action signature {Lj,L^ and {Li,Lq^ as the set of all test suites over an action signature 
respectively. 

For a given specification pQTS = {^Ao,Li,Lq,A^, we say that a test t is a test for safg, if it is 
based on the same action signature {Lj,L^ . Similar to before, we denote all tests for si^s^y {s^s) and 
all test suites hy respectively. 

Note that we mirrored the action signature for tests, as can be seen in Figure 3a and Figure 
respectively. That is, because we require tests and implementations to shake hands on shared actions. A 
special role is dedicated to quiescence in the context of parallel composition, since the composed system 
is considered quiescent if and only if the two systems are quiescent. 

We will proceed to define parallel composition. Formally this means that output actions of one 
component are allowed to be present as input actions of the other component. These will be synchro¬ 
nized upon. However, keeping in mind the mirrored action signature of tests, we wish to avoid possibly 
unwanted synchronization, which is why we introduce system compatibility. 


Definition 19. (Compatibility) Two pQTS = [S,sq,Li,Lq,A) , and = (S',5 o,LJ,Lq , A') are said 
to be compatible if nL^' = {5}. 

When we put two pQTSs in parallel, they synchronize on shared actions, and evolve independently 
on others. Since the transitions taken by the two component of the composition are stochastically inde¬ 
pendent, we multiply the probabilities when taking shared actions. 


Definition 20. (Parallel composition) Given two compatible pQTS ^ = [S,so,Li,Lq,A) and = 
{s' ,s'q,L\,Lq,A'^, their parallel composition is the tuple 


jz/ II .s/' 



where 

S" = Sx S', 
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4 = i^o,s'o), 

L'l = {LjUL'j)\{LoUL'o), 

A" = {{{s,t),p) e S" X Distr{L” x S”) 


p {a, (s',t')) 


Pa(a,s')Va(a,t') 
Pa (a, s') 

Va{a,t') 

0 

\ 


if a G L n L', where s s' A t t' 

if a G L\L', where 5 s' At = t' 
if a G L'\L, where s = s' At f' 

otherwise 


where pa G Distr{L,S) and Vq G Distr {L',S') respeetively. 

Before we parallel compose a test case with a system, we obviously need to define which outcome 
of a test case is considered correct, and which is not (i.e., when it fails). 

Definition 21. (Test case annotation) For a given test t a test annotation is a function 


a : ctraces{t) —)• {pass Jail} . 

A pair t = {t,a) consisting of a test and a test annotation is called an annotated test. The set of all such t 
is defined as T = | {ti,aJ^j;'} for some index set is called annotated Test Suite. If t is a test case for 
a specification we define the pioco test annotation 0 ^°™ : ctraces{t) —)• {pass Jail} by 




ioco 


((^) 


fail if 3ai G traces ^ Lg : aia! ijo AO\a\ ^ traces {jz/J ; 

pass otherwise. 


4.2 Test execution. 

By taking the intersection of all complete traces within a test and all traces of an implementation, we will 
define the set of all traces that will be executed by an annotated test case. 

Definition 22. (Test execution) Let f be a test over the action signature and the pQTS = 

[S,so,Li,Lq,A). Then we define 


exect (M) = traces (M) H ctraces (?). 


Example 23. Consider the specification of a shuffle music player and a derived test for it given in Figure 
Assuming we are to test whether or not the following two implementations conform to the specification 
with respect to pioco: 



shuffle? 
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Here .. .,pn £ [0,1] such that pj = 1. Now when we compose £/i^ with t in Figur^^ we can 
clearly see that every complete trace of the parallel system is annotated with fail, as it would also have 
been the case for classical ioco theory. However, if we now also consider and compose it with the 
same test t, every trace of the composed system would be given a pass label if we restricted ourselves to 
the annotation function and the output verdict. Note how every trace shuffle? ■ Song_i\ is given probability 
Pi for / = 1,... ,N. The only restriction we assumed valid for p\,... ,pi^ is that they sum up to 1 so a 
correct distribution for would be pi = and pi = ■ ■ ■ = Pn = This, however, should intuitively 
not be given the verdict pass, since it differs from the uniform distribution given in the specification 

4.3 Test evaluation 

In order to give a verdict of whether or not the implementation passed the test (suite), we need to extend 
the test evaluation process of classical ioco testing with a statistical component. Thus the idea of eval¬ 
uating probabilistic systems becomes two folded. On the one hand, we want that no unexpected output 
(or unexpected quiescence) ever occurs during the execution. On the other hand, we want the observed 
frequencies of the SUT to conform in some way to the probabilities described in the specification. Thus 
the SUT will pass the test suite only if it passes both criteria. We will do this by augmenting classical 
ioco theory with zero hypothesis testing, which will be discussed in the following. 

To conduct an experiment, we need to define a lengfh k gN and a widfh m G N firsf. This refers fo 
how long fhe fraces we wanf fo record should be and how many times we resef fhe machine. This will 
give us fraces 0 \,... ,Om G L^, which we call a sample. Addifionally, we assume fhaf fhe implemenfafion 
is governed by an underlying frace disfribufion H in every run, fhus running fhe machine m fimes, gives 
us a sequence of possibly m differenf frace disfribufions H = Hi,.. So in every run fhe implemen¬ 
fafion makes fwo choices: 1) If chooses fhe frace disfribufion H and 2) H chooses a frace a fo execufe. 
Consequenfly fhaf means fhaf once a frace disfribufion Hi is chosen, if is solely responsible for fhe frace 
Gi. Thus for i j fhe choice of a, is independenf from fhe choice of Gj. 

Our sfafisfical analysis is build upon fhe frequencies of fraces occurring in a sample O. Thus fhe 
frequency function will be defined as 

/rgg(6 >)(a)= 

m 

Nofe fhaf alfhough every run is governed by possibly differenf frace disfribufions, we can sfill derive 
useful informafion from fhe frequency funcfion. For fixed k,m gN and H, fhe sample O can be freafed 
as a Bernoulli experimenf of lengfh m, where success occurs in position / = 1,.. .m, if a = a,. The 
success probabilify is fhen given by P//. (a). So for given H, fhe expecfed value for a is given by 
^ T 4 L 1 Phi (<7)- Nofe fhaf fhis expecfed value is fhe expecfed disfribufion over if we assume 
if is based on fhe m frace disfribufions H. 

In order fo apply zero hypofhesis fesfing and compare an observed disfribufion wifh E^, we will 
use fhe notion of mefric spaces. This will enable us fo measure deviafion of fwo disfribufions. We 
will use fhe mefric space {L^,dist), where dist is fhe euclidean disfance of fwo disfribufions defined as 

dist{p,v) = ^JZaeLAp{(y)-v{c>)\^. 

Now fhaf we have a measure of deviafion, we can say fhaf a sample O is accepfed iifreq{0) lies 
in some disfance r of fhe expecfed value E^, or equivalenfly if freq{0) is confained in fhe closed ball 
Br ^E^^ = |v G Distr(f}^ \ dist ^v,E^^ < r|. Then fhe s&tfreq^^ (^r summarizes all sam¬ 

ples fhaf deviafe af mosf r from fhe expecfed value. 
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Figure 4: A probabilistic automaton representing a fair coin. 


An inherent problem of hypothesis testing are the type 1 and type 2 errors, i.e. the probability of 
falsely accepting the hypothesis or falsely rejecting it. This problem is established in our framework by 
the choice of a level of significance a G [0,1] and connected with it, the choice of radius r for the ball 
mentioned above. So for a given level of significance a the following choice of the radius will in some 
sense minimize the probability of false acceptance of an erroneous sample and of false rejection of a 
valid sample (i.e., at most a). 

r := inf |r | Pp (freq^^ (^Br ^ ^ > 1 - a|. 

Thus assuming we have m different underlying trace distributions, we can determine when an observed 
sample seems reasonable and is declared valid. Unifying over all sets of such H, we will define the total 
set of acceptable outcomes, called Observations. 

Definition 24. The acceptable outcomes of H with significance level a G [0,1] are given by the set of 
samples of length k G N and width m G N, defined as 

Obs :=freq^^ ^ ^ • 

The set of observations of £/ with significance level a G [0,1] is given by 

Obs{s^,a)= [J Obs{H,o^ . 

Hetrd(j2/,k)"' 

Example 25. Assume that the wanted level of significance is given by a = 0.05 and consider the proba¬ 
bilistic automaton in Figure |^representing the toss of a fair coin. Furthermore assume that we are given 
two samples of depth k = 2 and width m= 100. 

To sample this case, assume E is the adversary that assigns probability equal to 1 to the unique 
outgoing transition (if there is one) and probability 1 to halting, in case there is no outgoing transition. 
We take H = trd{E) and can see, that then pnia^bl) = pH{alc\) = ^ and /^//(cj) = 0 for all other 
sequences a. We define = {H\,... ,//loo)^ where Hi = ... = Hiqq = H. As we can see, we have 
= Ph- Since pn only assigns positive probability to alb\ and alc\, we get (Bripn)) = 
— r <freq{0) {alb\) < ^ +?■})■ One can show that the smallest ball, where this probability is 
greater or equal than 0.95 is given by the ball of radius f = -^. 

Thus a sample 0\, which consists of 42 times alb\ and 58 times alc \ is an observation, and a sample 
O 2 , which consists of 38 times alb\ and 62 times alc\ is not. 

Thus we can finally define a verdict function, that assigns pass when a test case never finds erroneous 
behaviour (i.e. wrong output or wrong probabilistic behaviour). 
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Definition 26. (Output verdict) Let {Li,L^ be an action signature and f = (t,a) an annotated test case 
over (L/,Lq). The output verdict function for t is the function vj : pQTS {pass Jail}, given for any 
pQTS M 

I pass if Va G exect {s^i) : a (a) = pass 
\fail otherwise 


Vf {s^i) = 


(Statistical verdict) Additionally let a G [0,1] and k,m G N and O G Obs{£/i\\t,a) C then the 

statistical verdict function is given by 


vf 


pass if O G Obs (j^s, CC) 
fail otherwise 


(Verdict function) For any given we assign the verdict 




pass if Vf (M) = vf {^/i) = pass 
fail otherwise 


We extend VA to a function VF : pQTS —{pass Jail}, which assigns verdicts to a pQTS based on a 
given annotated test suite by VF (^) = pass if for all f G T and VF =fail otherwise. 


5 Conclusion and Future Work 

We introduced the core of a probabilistic test theory by extending classical ioco theory. We defined the 
conformance relation pioco for probabilistic quiescent transition systems, and proved several character¬ 
istic properties. In particular, we showed that pioco is a conservative extension of ioco. Second, we have 
provided definitions of a test case, test execution and test evaluation. Here, test execution is crucial, since 
it needs to assess whether the observed behaviour respects the probabilities in the specification pQTS. 
Following 14 ], we have used statistical hypothesis testing here. 

Being a first step, there is ample future work to be carried out. First, it is important to establish the 
correctness of the testing framework, by showing the soundness and completeness. Second, we would 
like to implement our framework in the MBT testing framework JTorX, and test realistic applications. 
Also, we would like to extend our theory to handle T-transitions. Finally, we think that tests themselves 
should be probabilistic, in particular since many MBT tools in practice do already choose their next 
action probabilistically. 
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Appendix 

Below, we present the proofs of our theorems. 


Proofs 

Proof of Theorem 14. 

” <;= ” Let i^pioco and a G traces Our goal is to show out^. (a) C (a). 

For (a) = 0 we are done, sinee 0 C out^^ (a) obviously. 

So assume that there is b\ G (a). We want to show that b\ G outj^/^ (a). For this, let k=\G\ and 
H G trd sueh that Ph (cj) = 1, whieh is possible beeause a G traces {^/s) and both and £/s are 

non-probabilistie. The same argument gives us outcont{H,£/i,k) / 0, beeause a G traces (M)- 

Thus we have at least one H' G outcontfH,^i,k) sueh that Ph' {ob\) > 0. Let n G trace^^ (a) n 
Path* {s^s). Now PI' G outcont{H,£/s,k), beeause ^pioco by assumption and thus there must 
be at least one adversary E' G adv {safs,k+ 1) sueh that trd{E') = H' and {n • Dirac • b\s') > 0 for 
some s' G S. Henee E' (jt) {Dirac) Dirac {bl y)> 0 and therefore with s' G reach {last {n) ,b\) this yields 
b\ G out^^ (a). 

” ” Let £/i C/oco -s/s, ^ G N and H* G trd{s^sik). Assume that H G outcont {H*,£/i,k), then we 

want to show that H G outcont {H*,s^s,k). 

Therefore let E G adv {.s^i,k+\) sueh that trd{E) = H. If we ean find E' G adv {s^s:b + \) sueh that 
trd {E) = trd {E'), we are done. We will do this eonstruetively in three steps. 

1) By eonstruetion of H* we know that there must be E' G adv{£^s,k + \), sueh that for all o € l!' 
we have Ptrd{E') = Ph* ((^) = Ptrd{E) (<^)- Thus H* Qk trd{E'). 


2) We did not speeify the behaviour of E' for path of length k+\. Therefore we ehoose E' sueh that for 
all traees o € l!' and al G Lj we have Ptrd{E') {oal) = 0 = Ptrd{E) {oaf). 

3) The last thing to show is that trd{E) = trd{E'). Therefore let us now set the behaviour of E' for 
traees ending in outputs. Let a G traces {n^i), then assume a\ G out^^ (a) (if (a) = 0 we are done 
immediately) and beeause Eioco we know that a! G (cj). 

Now let p := Ptrd(E) {o) = Ptrd{E') (o) and q := Ptrd(E) equality of the traee distributions 

for traees up to length k we know that ^ < p < 1 and therefore there is a G [0,1] sueh that q = p - a. Let 
traces {r^s) H trace^^ (a) = {tti ,... , 7r„}. Without loss of generality, we ehoose E' sueh that 

,, , , , fa if / = 1 

E {Ki) {Dirac) = < 

I 0 else 

We eonstrueted E' G adv {rat's, k+ 1), sueh that for all a G we have Ptrd{E') {o) = Ptrd(E) {o) and thus 
trd {E) = trd {E'), whieh finally yields H G outcont {H* ,£/s,k). □ 

Proof of Lemma 15. Let s^s then for every H G trd {£Pi,k) we also have H G trd (i 24 ,k). So piek 

m G N, let H* G trd{£/s,m) and take H G outcont {H* ,£/i,m) C trd{£/i,m + \). We want to show that 
H G outcont {H*, rat's, m). 
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By assumption we know that H G trd + 1). In particular that means there must be at least one 
adversary E G adv (, m + 1) such that trd{E) = H. However, for this adversary, we know that H* 
trd {E) and for all a G L'”Lj we have Ptrd{E) (^) = 0 and by trace distribution inclusion trd (E) =H. Thus 
H G outcont {H *, m) and therefore Ep/oco -s/s- El 


Proof of Theorem 16. ” => ” Let Epioco ^s, fix m G N and take a trace distribution El* G trd 
To show that El* G trd (^,/n), we prove that every prefix of El* is in trd {£/s,tn), i.e. if H' Et: H* for 
some k G N, then H' G trd (^). The proof is by induction up to m G N over the prefix trace distribution 
length, denoted by k. 

Obviously H' G trd{£/i,0) yields both H' Eo H* and H' G trd{£/s)- Now assume, we know that 
H' Ejt H* for some k <m and El' G trd Furthermore let H" G trd (^,k + 1), such that H" E/t+i H*■ 
If we can show that H" G tr(i(^,k+ 1), we are done. 

With H' G trd{£/s,k), we take H'" G outcont {H' ,£/i,k) such that all traces of length k+l ending 
in an output action have the same probability, i.e. for all a G L'^Lq, we have Pff",{o)=PH"{o). By 
assumption £/i Epioco and thus H'" G outcont {H',£/s,k) C trd{£/s)- 

Let E G adv{£^s,k+ 1) the corresponding adversary such that trd{E) = H'". By construction, we 

in general 

have Ptrd(E) (<yal) = F//// (aa!) and Ptrd(E) {obi) = 0 / Ph" {obi) for all a G L^. We create yet another 
adversary, denoted by E' G adv {£/s,k + 1) such that for all a £ and a\ G Lq, we have Ptrd(E) (<7) = 
Ptrd(E') (ct) and Ptrd(E) (cta!) = Ptrd(E') Taking the sum over all probabilities of those traces yields 


I 


a\€L 


S 

O 


Ptrd(E) {Oa\) 1 CC, 


where a G [0,1] and consequently the remaining bit is covered by 

^ Ph" {obi) = a. 

bteLi 


The aim is now to set the behaviour of E' such that a G L^Lj has F//// (a) = Ptrd{E') {o)- We prove that this 
can indeed be done independently from a. The input enabledness gives that for all obi G traces {^/t), 
we also have obi G traces {^/s)- Assume Ph" (cj) = p and thus 

a = Y, Ph" (^^•) = Ph" {obil) + ...+Ph" {obnl) =pai + ...+ pa^ 

bteLi 

= Ptrd(E') {obil) + . . ■+Ptrd(E') {obnl) ■ 


However, since trd {E) Ejt H”, we also have Ptrd(E) (<7) = P- 

The last detail not yet specified about E' is the behaviour of paths of length k+l ending in an 
input transition. We demonstrate the choice of E' for pa^ = Ptrd{E') {obil), and denote the associated 
paths {ki ,..., 7r„} = trace^^ (a). Furthermore tt' := Ttipbi Isi- for some si- £ S, j = I,... ,1, which are 
reachable after tt, and distributions containing bl. Thus we want 
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pai = Ptrd{E'){(^bl) = ^PE>{n'i) 

i=l 

= L «) (M)/ t (^1?,^/,-) 

=p =:«! 

n / 

/= 1;=1 

'-V-" 

=1 

We can do the same for all a, for / = 1,..., ft). Note that the choice of the adversary does not depend on the 
chosen trace a but solely on the presupposed behaviour of PI”. Thus we have found E' G adv ( , ^ + 1) 

such that trd {E') = H”. Hence H” G trd (^2^)^+ !)> which ends the induction. Since this is possible for 
every m G N, we get Qpioco ending the proof. 


” -4= ” See Lemma 15 for the proof. In particular we do not even require input enabledness for £/s 
in this case. □ 


Proof of Theorem 17. Let sP Epioco and PS ^pwco 'S’ and £/ and be input enabled. By Theorem [T^ 
we know, that £/ Ejd pS. So let k G N and H* G trd {.s/,k). Consequently also H* G trd {PS,k) and thus 
the following embedding holds 


outcont {^7,H* ,k) C outcont {PS,H* ,k) C. outcont {^,H* ,k), 
and thus £/ Epioco 'S’. 


□ 





